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Abstract 

The growing demand for an increasingly skilled competitive workforce and the associated 
demand for change and responsiveness in the provision of technical vocational education 
and training (TVET) has led to the development of stronger links between New Zealand and 
the People’s Republic of China. A collaborative model programme project aims to improve 
TVET provision in the two countries through curriculum re-design, joint programme 
development, and the delivery of quality New Zealand qualifications in China. Because there 
is there is often a gap or disconnect between policy intent and classroom reality, a critical 
activity of the model programme project aims to identify and measure when, how, and where 
this disconnect occurs. This paper identifies the performance indicators that are used to 
measure learner and institutional success, highlights the strategies used to evaluate the 
learning environments created, and reports on the development and validation of a user- 
driven, flexible, internet-based, learning environment instrument for use in multi-national 
TVET settings. It argues that this instrument provides model programme stakeholders with 
sufficient data to understand, economically and efficiently, the actual effect of change at the 
point of delivery. 

Keywords: online learning; distance education; e-leaming; distance learning; technology- 
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Context 

There is international acknowledgement that the ability of individuals, industry, and nations to 
meet the competitive challenges of an increasingly global economy is irrevocably bound to the 
tertiary vocational education and training (TVET) policies that have been implemented to 
establish an adaptable, highly-skilled and knowledgeable workforce (Cheong & Lee, 2016; 
Alagaraja, Kotamraju, & Kim, 2014; Maclean & Lai, 2011; Powell & Solga, 2010). It is also 
acknowledged that these standards cannot be raised in national isolation. Participants in 
vocational programmes and further training need to have opportunities to transfer seamlessly 
between learning environments (including national and international educational institutions) to 
improve their employability and quality of life (Spottl, 2013). In this context, the People’s 
Republic of China and New Zealand are collaborating on initiatives for workforce development 
for, firstly, an ongoing extension of the workplace skills and abilities of employees and trainees 
and, secondly, a review of formal TVET courses and programmes (Clayton, 2015; Hutchings, 
Zhu, Cooper, Zhang, & Shao, 2009). 

In 2010, New Zealand and the People’s Republic of China began investigating strategies to 
develop stronger links between the two countries with the aim of developing the capabilities and 
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capacities of TVET teaching staff, and to support stronger institutional relationships in TVET 
programmes. At the same time, China’s changing attitude towards vocational training, the 
establishment of exemplary TVET institutions to demonstrate best practice in TVET education 
(model polytechnics) and a broader recognition that vocational study was an acceptable 
educational outcome, opened up partnership opportunities between the New Zealand Institutes of 
Technology and Polytechnics and the People’s Republic of China model polytechnics (Ministry 
of Education of the People’s Republic of China and the New Zealand Ministry of Education 
[MOE PRC/NZ], 2015). The New Zealand-China Vocational Education and Training Model 
Programme (known as “The Model Programme”) was agreed to as a commitment in the Strategic 
Education Partnership Arrangement, which was signed during Prime Minister John Key’s visit to 
Beijing in 2013. Since then, TVET institutions in New Zealand and the People’s Republic of 
China have actively collaborated to learn from each other’s experiences. This collaboration 
includes curriculum design, joint programme developments, and the delivery of quality New 
Zealand qualifications in China (MOE PRC/NZ, 2015). 

In periods of educational change and reform it has been noted there is often a gap, a disconnect, 
between policy-makers’ stated intentions and the actual experience of the learners when that 
policy is implemented (Ng, 2008). In other words, there is a gap between policy rhetoric and 
classroom reality (Atai& Mazlum, 2013). For example, transnational agreements between 
People’s Republic of China and English educational institutions promised economic efficiencies 
and large learner cohorts. However, these efficiencies and student numbers were affected by 
declining numbers of school leavers, expanding provision in the People’s Republic of China, and 
competition from other international providers (Zhuang & Amelia, 2012). Multi-nation initiatives 
such as the model programme project therefore have the potential for a disconnect between what 
is intended and what is experienced. 

This paper identifies techniques and strategies to manage any such disconnect. Firstly it reviews 
the performance indicators commonly used to measure learner and institutional success; 
secondly, it describes a framework measure performance in TVET learning environments; and 
finally, it describes the development and validation of an instrument that is aligned with the 
framework to achieve the purposes of the model programme. 

Framework for measuring performance 

Measures of performance play a dual role in educational environments. Firstly, they are used to 
assess whether the initiatives implemented meet the desired indicators or outcomes (Asif & 
Searcy, 2014). Secondly, they are used to examine processes and foster an environment of 
continuous improvement (Al-Turki & Duffuaa, 2003). 

Measuring outcomes 

Measuring outcomes (the performance of the proposed changes on actual learner success) is 
generally achieved with standardised performance indicators. For the model programme project, 
it was decided to use established New Zealand Tertiary Education Commission indicators. These 
include: 


• Successful completion of courses: This indicator shows the proportion of courses in a 
given year that are successfully completed. Courses are usually parts of qualifications 
and are sometimes referred to as ‘papers’. 

• Completion of qualifications: This indicator shows the proportion of students in a given 
year who complete a qualification. It provides a good indication of how successful 
tertiary providers are in helping their students to complete qualifications. 
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• Student progression to higher level study: This indicator shows the proportion of 
students in a given year who progress to study at a higher level after completing a 
qualification at Levels 1-4. It shows how tertiary providers help successful students 
move to higher levels of tertiary education. 

• Students retained in study: This indicator shows the proportion of students in a given 
year that complete a qualification or re-enrol at the same tertiary provider in the 
following year. 

(Tertiary Education Commission, 2016). 

Using these established measures allows the managers of model programmes to identify any 
disconnect between policy intention and reality, highlight areas that need to be addressed and, 
importantly, benchmark the success of their newly designed curricula against traditional, 
established New Zealand programmes. 

Measuring process 

Measuring process—the intended versus actual classroom experience of learners—is more 
complex and challenging. A number of factors (e.g., the tutor, the learner, the classroom, the 
workbooks, and the assessments) need to be considered. In essence, the participants’ actual 
learning experience will be determined jointly by the individual and their environment (Fraser, 
1983). This approach, where indicators are not generated by the person or their environment 
separately (but rather by their fit or congruence with each other) is based on a person- 
environment fit (P-E fit) theory (Edwards, Caplan, & Harrison, 1998). In educational settings, 
performance measures (referred to as learning environment instruments) have been created using 
P-E fit theory. These instruments are developed with the formula, B=f(P,E) where behaviour (B) 
is considered to be a function of (f), the person (P), and the environment (E) (Clayton, 2009; 
Fraser, 2001). For the model programme project it was decided that development of a TVET- 
specialised, internet-based, learning environment instrument to gather data on the perceptions of 
learners would provide the following benefits: 

• It enables programme designers to measure the actual impact of change at the point of 
delivery. 

• The data generated by the instruments enables stakeholders to identify potential reasons 
of misalignment during delivery. 

• Early identification of specific gaps between intention and experience, and rhetoric and 
reality, means adjustments and refinements can be implemented during delivery. 

Learning environment instruments can be used to explore the association between learner 
outcomes and the designed environment, investigate the differences between learners’ 
perceptions and tutors’ intentions, and determine whether learners are successful in environments 
that suit their preferences (Koopman, Teune, & Beijaard, 2012; Law, Meyer, 2011). The 
feedback generated from the data can be used to guide changes to physical spaces, facilitate 
improvements in actual practice, and monitor the effectiveness of educational innovations 
(Fraser, 2012). An instrument developed to investigate the learning environments created in 
TVET contexts through the model programme initiative would be a valued tool. This instrument 
is the focus of the next section. 

Instrument development and delivery 

Although the nature of the social environment or change that the psychosocial instruments might 
investigate can vary markedly, instrument developers take great care to ensure the instruments 
are soundly developed and rigorously tested (Fraser, 2012). Specifically, in the field of learning 
environment research, there is an accepted methodology for developing and validating 
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instruments. First, researchers identify the salient dimensions and individual items that relate to 
the field of study. Second, they ensure the individual dimensions and items adequately cover the 
three social climate dimensions identified by Moos (i.e., relationship, system maintenance and 
system change, personal development) (Moos, 1976). Finally, the instrument is field tested and 
validated (Li, Kong, & Chen, 2015; Walker & Fraser, 2005). 

Instrument development 

Although these three stages have been identified, they are neither linear nor independent. Indeed, 
the stages can be seen to be interwoven, interdependent, and overlapping. For example, it would 
be wise to refer to social climate dimensions when identifying salient scales, and field testing a 
draft instrument with a limited audience may be an integral part of individual item identification. 
To accommodate this integrated nature of instrument development, two research concepts— 
content and construct validity—are used. In content validation, researchers check against the 
relevant knowledge of the domain (in this study, this knowledge is TVET practices and 
techniques), and ensure all aspects pertinent to the domain are identified (Lovric et al., 2016). 
Construct validity (the degree to which inferences can legitimately be made from the research) 
requires ensuring that the scales and items measure the construct identified (i.e., convergent 
validity), and that individual scales measure only a single construct (i.e., discriminant validity) 
(Li et al., 2015; Trochim, 2006). 

Delivery 

The ready availability of cloud-based electronic databases and the development of intuitive user 
interfaces for these databases has enabled researchers with limited technical knowledge to 
generate web forms to collect responses from participants. These dynamic web forms connect 
with responsive databases, making it possible to centrally collect, store, manipulate, analyse and 
report the data generated from multi-nation sites (Wright, 2005). The process of internet-based 
learning environment instrument development and administration is illustrated in Fig. 1. 
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Figure 1 Digital learning environment instrument procedures 
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Planning and development of the instrument to measure 
performance in the model programme 

In the development of an internet-based learning environment instrument to investigate the effect 
of change driven by the model programme TVET initiative at the point of delivery, three key 
work-streams have been identified. 

Work-stream 1 

Workstream 1 identifies salient dimensions and items relating to the changes in teaching and 
learning as a result of introducing the TVET model programmes. These changes may relate to the 
curriculum, assessment design, approaches to teaching and learning, moderation and evaluation, 
and engagement with industry. Salient scales and items created are reviewed by all stakeholders 
of the environment to be investigated (managers, designers, teachers, and learners) and by 
external educational researchers to ensure the scales and items used are adequate, suitable, and 
relevant. 

Work-stream 2 

Workstream 2 establishes a flexible digital instrument development platform, with an integrated 
database. This flexible platform is based on the concept of modular build, which allows 
stakeholders to create, from a specified library of validated scales and items, an instrument to 
investigate identified aspects of the environment. The creation engine and associated instructions 
are designed to ensure sufficient dimensions are included in all of the instruments used to 
interrogate the three general categories of socially constructed environments (i.e., personal 
growth; relationships; and systems, maintenance and change). This platform provides 
stakeholders with the functionality to design instruments to investigate specific changes to 
TVET, it allows them to make the instrument available to participants in multiple locations, and 
it creates a secure space for data to be gathered, securely stored, structured, retrieved, analysed, 
and compared. 

Work-stream 3 

Workstream 3 covers field testing and analysis. 

Stakeholders can: 

• review and comment on the scales and items 

• test the instrument generation and deployment environment 

• assist with piloting the internet-based instruments 

• review the efficiency of the databases 

• evaluate the validity of the process to identify the impact of change at point of need 

• validate the content and construct validity of the instruments. 

Implementation 

Two common strategies will be used to identify the salient scales and individual items that will 
be used to construct the TVET model programme provision instrument. First, guided by 
previously validated scales and relevant items, model programme designers from New Zealand 
and the People’s Republic of China will conduct an extensive literature review of current and 
emerging practices in TVET courses and programmes in both countries. Second, this literature 
review will be enhanced by extensive engagement with TVET teaching specialists, TVET 
educational researchers, and past and present TVET learners to confirm current practice and 
emerging trends. These two strategies will ensure that the dimensions, scales, and individual 
items that TVET stakeholders regard as salient, are covered (Fraser, 2012; Walker & Fraser, 
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2005; Fraser, 2001). It is anticipated five broad themes will influence the identification and 
selection of specific scales and items. 

These themes are: 

1. learner-physical environment interaction (the extent to which the student feels 
comfortable and enjoys learning in the physical space provided) 

2. learner-learner relationships (the extent to which students work together, know, help, 
support and are friendly to each other) 

3. learner-tutor relationships (the extent to which the tutor guides students in their learning 
and provides sensitive, ongoing, and encouraging support) 

4. learner-resource interaction (the extent to which class activities are clear, well organised, 
stimulating, and assist student comprehension) 

5. learners’ reflection activities (the extent to which reflective activities are encouraged and 
how students enjoy learning and participating in this environment). 

When building a digital instrument development platform, all of these scales and their associated 
items will be arranged as assets or modules, in the appropriate social dimension categories, and 
in an interactive web form. These dynamic web forms will be designed with intuitive “what you 
see is what you get” (WYSIWYG) software applications that are seamlessly integrated with an 
intuitive database (Clayton, 2009). Using these modular built web forms and integrated database 
software brings a number of advantages: 

• TVET learning environment instruments and associated databases can be created quickly 
and efficiently. 

• Instruments can be made available immediately and across distributed locations. 

• Some data-entry activities (which are often tedious) are eliminated, providing greater 
assurance that the data is free from common entry errors. 

• Because there is no separate data-entry phase, tabled results are available for analysis 
immediately after the data collection phase. 

• The costs in terms of both time and money by presenting the instrument on the web are 
low, compared with costs associated with conventional surveying methods (Liu & 
Clayton, 2016; Wright, 2005). 

During the field testing and analysis phase, a range of instruments will be developed and tested 
with a sample of the target population. This testing will ensure the instruments are economical in 
terms of the time needed for participant completion and stakeholder administration. The data 
generated during this phase will be reviewed and statistically analysed to determine the internal 
consistency and discriminant validity of each of the scales used (Lovric et al., 2016; Li, Kong, & 
Chen, 2015; Trochim, 2006). Tentative conclusions about the effects of the change will be drawn 
from the data and provided to stakeholders for review and comment. All stakeholders 
participating in this field-testing phase will be asked to provide feedback on the efficiency and 
effectiveness of the instrument development platform, the data collected, and the reports 
generated. 

The project team has created a demonstrator environment to review the technical requirements 
and to confirm the processes to create the instruments. This demonstrator environment is within 
an established learning management system. It is based on an editable web form that has three 
sections: relationship, system maintenance and system change, and personal development. The 
web form uses standard questionnaire functionality, that enables a responsive database (i.e., one 
for each instrument) to be generated as the instrument is built. Tentative scales and items, which 
are based on previous learning environment research, have been created as web assets and are 
aligned with the identified dimensions. These web assets can be added to the identified sections. 
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When each section contains at least one scale, the instrument can be published. The next phase of 
development is to work with TVET specialists to confirm scales and items. Test pilot instruments 
will then be created and made available to TVET learners in the people’s Republic of China and 
New Zealand. 

Discussion 

In responding to government demands for a highly skilled workforce, TVET institutions in New 
Zealand and the People’s Republic of China are actively collaborating to learn from each other’s 
experience. One of the outcomes of this collaboration will be to establish an infrastructure to 
design, develop, and deliver joint model programmes. To ensure the success of model 
programme provision, and to identify and address the potential disconnect between intention and 
reality, decision makers in both countries must have timely, reliable evidence on the impact of 
the model programme on learner and institutional achievement. They need to have the right 
information, at the right time, to make the right decisions. This paper argues for a framework— 
supported by measures of outcomes and process—for gathering this information. 

Outcome measures 

Outcome measures, which are based on established performance indicators, enable decision 
makers to benchmark the success of their newly designed curricula against traditional, 
established TVET programmes. However, because these quantitative measures focus on the 
outcomes achieved, they can be considered to be post-event indicators. Although they identify a 
disconnect between intention and experience, the measures do not identify when this disconnect 
actually occurred. In essence, the data generated by outcome measures cannot adequately 
measure the details of the learning process or the impact of change in TVET provision on a 
participant’s engagement. 

Process measures 

Process measures acknowledge that both the environment and its interaction with personal, 
individual characteristics are potent determinants of human behaviour and, ultimately, success. 
Learners form, and hold, persistent views on the TVET learning environment they engage with, 
and these established views affect the way they engage with the learning activities, materials, and 
assessments. Identifying these views at the point of delivery provides researchers with “during- 
the-event” indicators. These indicators help to identify when a disconnect occurs, and provide 
insights into how this disconnect can be addressed during delivery. 

Developing a validated psychosocial instrument and measuring students’ perceptions of the 
effect of a specific model programme change on their learning at the point of delivery provides 
managers, designers, tutors, and learners with a consistent method of gathering, sharing, and 
analysing data. This consistency enables model programme partners to make informed decisions 
on the type and timing of the adjustments needed to ensure the change affects TVET provision in 
the manner intended. 

It appears logical that the features that are successfully explored in learning environment research 
(the perceptions of learners of the environment and the social and psychological factors) will be 
of equal importance to research in TVET environments. Therefore, the development of an online 
perceptual measure investigating aspects of the TVET environments is timely, and can make a 
significant contribution to teaching, learning, and research in this area. 
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Four recommendations follow. 

1. TVET educators who are developing joint courses for delivery should be careful to 
ensure participants (both learners and tutors) are comfortable in, and benefit from, the 
learning environment. The availability of an interactive modifiable TVET instrument, 
and speedy access to data generated by its application, will highlight the practices that 
assist learners and tutors to adjust to the changing environment, and those that create 
barriers to learning. 

2. The availability of changeable web assets in the TVET instrument will enable 
researchers to target specific changes and economically monitor these changes for 
effectiveness. 

3. Validating the effectiveness of the instrument in two countries that have significant 
cultural differences will help to demonstrate the efficiency and effectiveness of the 
instrument across cultures and countries. 

4. As the instrument progresses through validation procedures it could contribute 
significantly to the larger research field of learning environment studies and TVET 
provision. 

Conclusion 

The growing need for a skilled competitive workforce, the demand for change and 
responsiveness in TVET provision, and the growing demand for continually upskilling a lifelong 
learning workforce are making vocational education and industry training increasingly viable, 
attractive, cost-effective, and valued. This continued growth of TVET provision needs to be 
matched by a similar growth in educational research that focuses on the specific learning 
environments created for these TVET worlds. The development, validation, and refinement of a 
framework that measures the efficiency of TVET provision will go some way to address these 
research needs. 
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